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PREFACE 


This  paper  documents  a  joint  research  effort  performed  under  a  Cooperative 
Research  and  Development  Agreement  (CRDA-93-014-AL-1)  between  Northrop 
Grumman  and  Armstrong  Laboratory’s  Aircrew  Training  Research  Division  (AL/HRA). 
The  work  was  conducted  with  Ethicon  Endo-Surgery,  Inc.  to  develop  prototype 
simulation  systems  to  help  train  surgeons  in  triage  assessment  and  procedures  for 
battlefield  casualty  management.  This  effort  is  documented  under  Work  Unit  1 123-B2- 
15,  Synthetic  Environments  for  Air  Warrior  Training  Research;  Work  Unit  Monitor  was 
Dr  Herbert  H.  Bell. 


Ill 


Medicine  Meets  Virtual  Reality:  4 

Health  Care  in  the  Information  Age:  Future  Tools  for  Transforming  Medicine 

17-21  January  1996 

Integrated  Medical  Environments  and  the  Virtual  Reality  Patient 

Bettina  A.  Babbitt,  Ph.D. 

Northrop  Grumman  Corporation 

Jack  Stubbs,  M.S. 

Ethicon  Endo-Surgery,  Inc. 

H.  Barbara  Sorensen,  Ph.D. 

Herbert  H.  Bell,  Ph.D. 

Peter  Crane,  Ph.D. 

USAF,  Armstrong  Laboratory 

ABSTRACT 

Training  today's  surgical  team  has  much  in  common  with  training  the  operators 
and  maintainers  of  today's  aerospace  systems.  In  both  cases,  individuals  must 
assess  the  situation,  perform  high  skilled  activities,  and  rapidly  adjust  as 
situations  change.  The  military  and  aerospace  communities  have  successfully 
developed  and  implemented  tools  and  techniques  for  such  training.  This 
investment  has  produced  significant  advances  in  training  system  design, 
realistic  training  simulations,  and  artificial  intelligence  (cognitive  task  analysis 
and  expert  systems).  These  same  technologies  will  also  improve  training  for 
biomedical  personnel. 

Northrop  Grumman  and  Ethicon  Endo-Surgery,  with  assistance  from  the 
Armstrong  Laboratory,  are  developing  prototype  simulation  systems.  This 
application  will  help  train  surgeons  in  triage  assessment  and  procedures  for 
battlefield  casualty  management.  The  goal  is  to  develop  an  integrated  training 
system  built  around  three  major  components:  (1)  A  Biomedical  Instructional 
Systems  Decision  Support  System  (B  ISD  DSS)  to  characterize  training  tasks, 
set  instructional  objectives  and  performance  standards.  (2)  A  simulation  based 
on  the  Preceptor  tm  virtual  reality  patient  to  support  training  for  battlefield 
wound  diagnosis  and  laparoscopic  surgery.  (3)  An  intelligent  tutoring  system 
hosted  on  PreCeptor  tm  to  teach  decision  making  skills  for  battlefield  wound 
diagnosis  and  subsequent  laparoscopic  procedures. 

Prototype  development  is  currently  underway  using  PreCeptor  tm  .  The 
prototype  focuses  on  diagnostic  and  therapeutic  procedures.  Instructional 
content  focuses  on  diagnosis,  pre-operative  preparation,  laparoscopic  surgery, 
and  post-operative  care. 
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INTRODUCTION 


Several  training  technologies  from  the  military  and  aerospace  are  identified  for 
their  direct  application  to  surgical  environments.  Specifically,  decision  support 
systems,  conceptual  graph  analysis,  and  expert  systems  are  discussed  as  to 
their  relationship  to  training  in  virtual  reality.  With  new  and  adapted 
methodologies  from  the  field  of  artificial  intelligence  (cognitive  task  analysis  and 
expert  systems)  it  is  now  possible  to  teach  both  surgerical  skills  and  decision 
making,  as  applied  to  laparoscopic  surgery.  The  virtual  reality  patient, 
PreCeptor  tm,  will  be  developed  to  host  content  for  battlefield  triage  wounds. 
Previous  PreCeptor  tm  content  was  associated  with  diagnosis,  pre-operative 
preparation,  laparoscopic  surgery,  and  post-operative  care.  Virtual  reality  as  a 
training  medium  for  surgery  holds  promise  to  enhance  and  expand  traditional 
ways  of  teaching  medicine. 

MILITARY  AND  AEROSPACE  INVESTMENT  IN  MEDICINE 

Prior  to  World  War  II  the  need  to  create  a  structured  approach  to  develop  and 
implement  training  systems  was  recognized,  but  no  standardized  approach 
existed  until  the  1960's.  At  that  time  procedures  known  as  Instructional 
Systems  Development  (ISD)  were  established  to  provide  a  more  structured 
approach  to  training.  These  ISD  procedures  have  been  adopted  by  the  United 
States  Army,  Navy,  and  Air  Force  for  weapon  systems  such  as:  M-1  Battle  Tank, 
Polaris  Missile,  and  B-2  Bomber.  Over  time,  additional  methods  and  models 
were  created  to  support  ISD  throughout  all  phases  of  the  weapon  acquisition 
process  (Babbitt,  Bolstad,  Craft,  Semple,  and  Sparks,  1989). 

Early  versions  of  media  selection  models  did  not  provide  estimates  of  training 
effectiveness.  Computational  power  was  limited  for  ISD  models,  and  models 
were  designed  to  analyze  small  increments  of  the  ISD  process.  New  technical 
approaches  were  developed  to  select  instructional  media  early  In  the  ISD 
process.  Evolution  of  media  mix  allocation  models  expanded  the  methodology 
so  that  metrics  of  both  cost  and  training  effectiveness  were  included  by  the 
1970's  (Sticha,  Blacksten,  Knerr,  Morrison,  and  Cross,  1986). 

During  the  1980's  military  and  aerospace  interest  In  computer  based  training 
expanded  to  include  technologies  from  artificial  intelligence.  Initially  the 
artificial  intelligence  (Al)  applications  were  simulation  based,  but  did  not 
incorporate  training.  For  example,  ADEPT  aided  battlefield  situation 
assessment  analysts  by  providing  tactical  interpretations  of  intelligence  sensor 
reports.  ADEPT  was  a  research  prototype.  Another  research  prototype 
developed  by  Science  Applications,  Inc.  was  known  as  Ocean  Surveillance. 

This  system  helped  naval  personnel  aboard  a  surveillance  ship  to  determine  a 
remotely  sensed  vessel's  destination  and  mission.  Ocean  Surveillance  used 
information  about  the  vessel's  correlated  tracks,  history,  location,  and  status  to 
determine  its  likely  destination,  arrival  time,  and  probable  mission. 

In  the  1990's  the  Human  Systems  Advanced  Technology  (HSAT)  project. 
Cooperative  Research  and  Development  Agreement,  between  Northrop 
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Grumman  and  USAF,  Armstrong  Laboratory,  started  adapting  ISO  and  Al 
training  technologies  for  application  to  aviation.  Subsequently,  these 
technologies  were  applied  to  surgical  simulation  in  conjunction  with  Ethicon 
Endo-Surgery.  The  technologies  have  roots  in  military  training,  instructional 
systems  development  (ISO),  artificial  Intelligence  (Al),  and  simulation. 

ISO  AND  Al  TECHNOLOGY  MIGRATION  TO  VIRTUAL  REALITY 

Three  technologies  which  have  been  used  in  the  HSAT  project  to  develop 
aerospace  training  systems  are  being  applied  to  medical  training.  The  three 
technologies  are:  (1)  decision  support  systems,  (2)  conceptual  graph  analysis, 
and  (3)  expert  systems. 

Decision  Support  Systems  -  Instructional  Systems  Development  (ISD) 
Decision  Support  Systems  (DSS)  have  traditionally  been  used  by  the  military 
and  aerospace  for  front-end  analysis  of  training  requirements.  According  to 
Babbitt,  Sorensen,  and  Stubbs  (1995),  ISD  decision  support  systems  are  an  aid 
that  helps  the  human  decision  maker  generate  questions  and  answers  about  a 
proposed  training  system  before  it  is  designed  and  programmed.  Decision 
support  systems  answer  questions  such  as  what  tasks  must  be  trained,  and 
what  performance  levels  must  tasks  be  trained  to. 

The  Joint  Service  Instructional  Systems  Development/Aircrew  Training 
Decision  Support  System  (JS  ISD/AT  DSS)  was  developed  as  a  front-end 
analysis  tool  to  structure  an  aircrew  training  system,  it  includes  the  capability  to 
identify  clusters  of  tasks  for  cognitive  task  analysis,  and  assists  instructional 
designers  in  the  design  and  development  of  aircrew  training  systems  (Babbitt, 
Sorensen,  Vreuls,  Obermayer,  Muckier,  and  Gordon,  1994).  Since  JS  ISD/AT 
DSS  contains  many  generic  modules  and  high  degrees  of  decision  making  are 
required  both  in  aviation  and  medicine,  it  was  decided  to  convert  the  aircrew 
training  decision  support  system  into  a  biomedical  decision  support  system. 

The  Biomedical  Instructional  Systems  Development  Decision  Support  System 
(B  ISD  DSS)  contains  similar  modules  found  in  JS  ISD/AT  DSS.  These 
modules  were  adapted  to  a  surgical  application.  Of  particular  interest  is  the 
cognitive  rating  scale  where  cognition  is  scaled  on  22  cognitive  abilities. 
Cognitive  scaling  allows  identification  of  high  level  decision  making  tasks 
associated  with  surgeon  activities  before,  during,  and  after  surgery.  Figure  1, 
Cognitive  Abilities  Rating  Scale  Adapted  from  B  ISD  DSS,  shows  a  sample  of 
10  of  the  22  cognitive  abilities  mentioned  above.  Each  cognitive  ability  is  rated 
on  a  scale  from  1  through  7  using  the  automated  decision  support  system. 
Cognitive  abilities  identified  on  this  scale  during  task  analysis  are  clustered  by 
scale  points  between  1  and  7.  Cut  off  scores  of  6  and  7  have  been  used  to 
identify  and  cluster  tasks  for  further  analysis  known  as  cognitive  task  analysis. 
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Figure  1. 

Cognitive  Abilities  Rating  Scale 
Adapted  from  B  ISO  DSS 


Cognitive  Task  Analysis  -  Cognitive  task  analysis  focuses  on  the 
knowledge  necessary  to  select  tasks  which  represent  mental  activity  such  as 
problem  solving,  whereas,  task  analysis  focuses  on  the  procedures  needed  for 
successful  task  performance.  Tasks  requiring  extensive  decision  making  are 
further  analyzed  using  cognitive  task  analysis  (CTA).  CTA  has  been  used  in 
aviation  to  build  intelligent  flight  simulation  scenarios.  It  became  apparent  that 
CTA  had  a  role  to  play  in  medicine  as  well  due  to  the  similarities  between 
aircrews  and  surgical  teams  where  high  levels  of  decision  making  are  required 
within  time  compressed  conditions  (cockpit  and  surgical  suite).  Cognitive  task 
analysis  represents  knowledge  in  object  hierarchies.  Object  hierarchies  display 
relationships  among  higher-level  objects  and  lower  level  objects.  Several 
cognitive  models  exist  such  as  Bloom’s  cognitive  taxonomy  where  the  cognitive 
domain  is  divided  into  six  main  categories;  knowledge,  comprehension, 
application,  analysis,  synthesis,  and  evaluation  (Fleishman  and 
Quaintance,1984). 

Another  version  of  CTA  comes  from  Papantonopoulos  and  Salvendy  (1993), 
where  cognitive  task  allocation  can  be  defined  as  the  phase  in  the  design  of 
prirnarily  cognitive  tasks  determining  which  functions  are  to  be  performed  by  the 
various  system  components.  They  refer  to  the  human  operator  or  the  computer 
in  their  application.  In  our  particular  project  the  system  components  include  the 
surgeon  (hunian  operator),  equipment  (includes  hardware  and  software), 
laparoscopic  instruments,  and  simulated  anatomical  structures  of  the  patient. 

Several  methods  have  been  created  to  conduct  a  formal  CTA.  These 
methodologies  have  been  adapted  from  the  field  of  artificial  intelligence,  and 
are  known  as  knowledge  elicitation  and  knowledge  representation.  Conceptual 
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graph  analysis  is  a  knowledge  elicitation  and  representation  tool  for  general 
knowledge  acquisition  that  supports  the  instructional  designer.  It  is  an 
important  way  to  represent  decision  making. 

Conceptual  graph  analysis  (CGA)  is  used  to  represent  knowledge  structuring 
around  large,  complex  databases.  It  provides  a  means  of  representing 
concepts  and  their  relationship  to  support  the  performance  of  human  cognitive 
activities.  Instructional  designers  use  conceptual  graph  analysis  to  identify  the 
information  content  of  the  system,  represent  the  format  of  the  information  within 
the  system,  and  the  means  by  which  the  user  will  access  the  information. 
Conceptual  graph  analysis  users  perform  a  systematic  querying  technique 
called  question  probes  with  a  graphical  notation  system  called  conceptual 
graph  structures. 


According  to  Gordon,  Kinghorn,  and  Schmierer  (1991),  conceptual  graph 
analysis  represents  three  types  of  knowledge:  (1)  verbalizable  semantic 
knowledge,  (2)  verbalizable  rule/procedure  knowledge,  and  non-verbalizable 
implicit  knowledge.  Through  knowledge  elicitation  all  three  types  of  knowledge 
are  represented  in  a  conceptual  network  containing  nodes  and  arcs.  Content  of 
the  nodes  is  classified  as  either  a  Concept,  State,  Event,  Style,  Goal,  or 
Goal/Action.  Arcs  between  nodes  represents  the  relationship  between  nodes. 
Graphs  tend  to  have  subclusters  containing  certain  taxonomic  structures.  Table 
1  is  an  illustration  of  a  CGA  adapted  from  Gordon,  Kinghorn,  and  Schmierer, 
(1991).  Table  1  represents  node  categories  and  arc  labels  for  goal/hierarchy 
structures.  Conceptual  graph  analysis  representation  could  be  applied  to 
surgery.  For  example,  the  source  node  might  have  the  goal  to  rotate  a 
laparoscopic  stapler  clockwise  and  counterclockwise.  The  arc  would  represent 
the  reason  for  the  stapler  rotation  such  as  to  observe  that  there  is  no  other 
tissue  in  the  stapler.  The  terminal  node  goal  might  represent  stapler  firing  the 
staples  without  the  ureter  being  in  the  stapler. 


Kod& 


Cold,  GoaVActlon  VSii'jji^fteagoni  Goal 

Goal/Acticm^F 


Source  ttode 


Goid,  GoaVAoSonjillJIlll.  Means 
Event,  Slate,  ^Initiate 


■  Goal/Actton 
Goat  ^ 
GOatActfOfi 


Ooal/Acquitfofi  Evanir 


Arc  Dafinlttont 

Tmnfnal  iKxie  !a 
t^aaaon,  modva^or 
auparordioaia  noda 
source  noda. 

Scairce  node  Initratee 
or  tdggiars  tha  terminal 
nodes 

Source  node  ceuaes 
or  enables  lenntnia 
nodaorpracadae 
terminal  node^ 


Table  1. 


Node  Categories  and  Arc  Labels 
for  Goal/Hierarchy  Examples 
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Expert  Systems  -  Conceptual  graph  analysis  forms  the  foundation  for 
programming  expert  system  surgical  simulation.  Node,  arc,  and 
goal/hierarchies  formed  into  clusters  to  describe  cognitive  structures  are  used  to 
construct  a  rule  based  system.  Cognitive  strutures  are  also  used  to  design 
instructional  content.  The  domain  under  investigation  was  for  a  biomedical 
version  of  a  surgical  training  system.  In  this  system  the  expert  system  performs 
the  same  role  as  the  surgical  expert. 

Figure  2  displays  a  diagram  of  the  knowledge  engineering  process  where  the 
surgeon’s  knowledge  is  transferred  to  a  computer  program.  To  make  the  expert 
system  intelligent  requires  high-quality,  specific  knowledge  about  the  surgerical 
domain.  The  knowledge  is  obtained  through  medical  texts.  Interviews,  video 
tapes,  and  observation.  The  computer  program  is  designed  to  solve  problems 
in  much  the  same  way  as  the  expert  surgeon. 


Queries,  problems 


Expert  System 


Answers,  solutions 


Figure  2. 

Knowledge  Engineering  Transfers  Surgical  Knowledge  to 
Expert  System  Software  Program 

The  computer  program  uses  an  expert  system  shell  to  build  the  laparoscopic 
surgery  domain  rule  base.  The  expert  system  shell  contains  a  database 
structure  to  hold  the  laparoscopic  surgical  data  which  the  program  rules  access 
and  operate  on.  An  inference  engine  processes  the  rules  to  generate  surgical 
answers  and  solutions.  Interfaces  are  provided  to  enable  rule  modification,  and 
to  test  rule  sets.  In  summary,  the  expert  system  contains  the  laparoscopic 
surgery  database,  an  inference  engine  with  surgical  rules  (obtained  from  CTA), 
and  an  interface  for  the  programmer. 

Expert  system  shells  are  interfaced  with  a  dynamic  model  of  the  anatomy. 
Anatomical  simulation  requires  modeling  interrelationships  between  anatomical 
objects.  System  responses  to  external  events,  and  sequences  of  interactions  to 
be  programmed  are  described  in  software  event  flow  diagrams.  Event  flow 
diagrams  are  used  to  describe  surgical  vignettes  of  typical  interaction 
sequences.  Multiple  software  programs  are  interfaced  in  order  to  create  an 
intelligent  surgical  simulation. 
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INTELUGENT  VIRTUAL  PATIENT  ENVIRONMENT  (IVPE) 

IVPE  -  Signifies  the  next  generation  surgical  simulator  with  application  to 
battlefield  wounds.  Several  medical  lessons  learned  through  Operations 
Desert  Shield  and  Desert  Storm  called  for  improving  medical  readiness 
according  to  the  “Medical  Readiness  Strategic  Plan:  1995  -  2001."  The  plan 
calls  for  the  inclusion  of  medical  requirements  in  all  wargaming  activities.  DoD 
has  determined  that  shortfalls  exist  in  medical  readiness  training  in  the  Active 
and  Reserve  medical  forces  across  all  military  services.  IVPE  provides  an 
opportunity  for  specialty  surgical  skill  sustainment  within  a  battlefield  context. 

SUMMARY 

Technologies  identified  for  Instructional  Systems  Development  Decision 
Support  Systems  (ISD  DSS),  Cognitive  Task  Analysis,  Conceptual  Graph 
Analysis,  Expert  systems,  an  PreCeptorrM  all  played  a  role  in  bringing  lessons 
learned  into  surgical  simulation. 

In  summary,  the  B-2  Division,  Northrop  Grumman,  used  an  aircrew  version  of 
the  decision  support  system  to  analyze  upgrades  to  the  B-2  bomber  aircrew 
member  training.  PreCeptorrM  development  for  Ethicon  Endo-Surgery  included 
analysis  of  surgeon  abilities  and  decision  making  with  the  cognitive  rating 
scales  found  in  B  ISD  DSS.  PreCeptorrM  lessons  learned  were  useful  in 
conceptualizing  the  structure  of  vignettes  which  will  be  hosted  on  the 
PreCeptorrM  surgical  simulator.  A  form  of  cognitive  task  analysis  known  as 
conceptual  graph  analysis  combined  with  expert  systems  was  used  to  structure 
an  F-16  intelligent  flight  simulation  for  the  USAF,  Armstrong  Laboratory,  and 
Northrop  Grumman.  Technologies  combined  in  new  applications  with  previous 
lessons  learned  will  be  employed  on  IVPE. 
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